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Abstract 

Meaningful learning based on PBL is new learning strategy. Compared to traditional learning strategy, the 
meaningful learning strategy put the student in center of the learning process. The roles of the student in the 
meaningful learning strategy will be increased. The Problem-based Learning (PBL) model is considered the most 
productive way to encourage development of higher order thinking. In this paper we introduce application of an 
innovative approach to complex, adaptive, and computer-mediated assessment of HOT skills development of 
individual students. The complexity of assessment is expressed by forming the combined creative and analytical 
assessments of HOT skills. Analytical assessments skills are focused on evaluating, selecting, contrasting, 
comparing, and ordering. Creative Assessment adaptation is provided for individual students as for a study group. 
Creative assessments skills are focused on originality, efficiency, flexibility, problem finding, and elaboration. 
Results showed that PBL changes the learning process in many ways. It changes the relationship between 
learner and teacher, and has an impact at the leaner level. The class has been re-organized and re- evaluated 
according of the principles of PBL. From the teacher’s perspective, PBL demands fundamental and deep 
reflection on work practices. 

Keywords: Problem-Based Learning, Higher-Order thinking, personalized learning, analytical thinking, critical 
thinking. 

1. Introduction 

PBL stands for both Problem-Based Learning and Project-Based Learning. PBL has proven to be a successful 
educational strategy in various fields. In computer science course has been developed new educational concept 
of PBL within higher education especially academic college for education. The goal of the proposed approach is 
developing HOT skills of students through adaptive problem-based learning. Adaptability of PBL is expressed 
by changes in fixed instructor assessments by crossing from one phase of PBL to another, choice of control tests 
and problems for students, and formation of heterogeneous collaborative groups. It should induce the students to 
develop HOT skills and collaborative skills through PBL. Adaptability of PBL is aimed at personalization of 
learning. 

There are two distinct types of HOT skills: analytical and creative thinking skills. Analytical or logical 
thinking skills allow critical thinking and help select the best alternative. Analytical thinking skills are: ordering, 
comparing, contrasting, evaluating, and selecting. Creative thinking skills are needed for problem solving. The 
creative thinking skills are problem finding (identifying the problem), efficiency (producing many ideas), 
flexibility (producing a broad range of ideas that characterizes flexible understanding), originality (producing 
uncommon ideas), and elaboration (developing ideas) (Hmelo-Silver, 2004; Bednarz, 2011; Cottrell, 2011; 
Cottrell, 2013). 

According to the proposed approach development HOT of students during study of subject by the class 
is realized through two phase process of Adaptive PBL. The course knowledge students acquire during the first 
phase. The attention on first phase is devoted by development analytical HOT skills of students. Fixed initial 
assessments of analytical HOT skills are set. After this phase analytical HOT skills of every student are 
examined by the personal control questions based on course material. It allows to assess development of the 
analytical skills. Next, adaptive collaborative group formation by taken into account of the assessment results of 
individual students is realized (Tilchin & Raiyn ,2015). 

The second phase of Adaptive PBL is fulfilled by the collaborative groups through problem solving 
process based on course material. The main attention on this phase is devoted by development creative HOT 
skills and collaborative skills of students. After the second phase developing HOT skills and collaborative skills 
of individual students are examined. Choice of control questions for examination of analytical HOT skills 
development is caused by the assessments of the skills after first phase. Development of creative HOT skills of a 
student is examined by control problems. It allows to assess development of the personal creative skills. 

2. Methodology 

Adaptive PBL directed towards HOT skills development is realized by the following steps: 

A. Preparation of the needed instructional materials for a course in computer science 

B. Assessment of developing analytical skills of the students after the first phase of PBL 

C. Building skill heterogeneous collaborative groups of the students 
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D. Adaptive formation of fixed assessments of HOT skills for second phase of PBL 

E. Assessment of HOT skills development of students after second phase of PBL 

F. Assessment of collaborative skills of students 

A. Preparation of the needed instructional materials for a course 

Preparation of the needed instructional materials for a course includes: 

• The forming of a list of course topics 

• Formulation of course requirements from a student with regard to development of HOT skills. After 
studying a course a student should have developed HOT skills and collaborative skills. Thereby, he (she) 
should be able to apply effectively acquired knowledge through taking creative solutions during solving of 
the proposed instructional problems, and explain cause-and-effect relationships. 

• The forming of a list instructional problems covering all learning outcomes and HOT skills 

• Setting fixed initial assessments of analytical HOT skills. Difference of the fixed initial assessments of the 
skills expresses the preference of an instructor in the development of certain skills during the first phase of 
Adaptive PBCL. An example of setting the fixed initial assessments is represented by Tablel. The 
“Evaluating” skill has the greatest need for development since they provide critical thinking. Consequently, 
the maximum fixed initial assessment is set for this skill. 

• The forming of control questions and instructional problems for examination of HOT skills of students. The 
instructor composes: a list of control questions which should examine analytical HOT skills of students after 
first phase of Adaptive PBL; a list of control questions and problems for examination of analytical and 
creative HOT skills of students after second phase. 

B. Assessment of developing analytical skills of students after the first phase of PBL 

A student’s analytical HOT skills are assessed by an instructor through the answers given by a student 

to control questions from the individual list. An instructor assessment of a student answer to a question varies 
from zero to the fixed initial assessment of analytical skill corresponding to a question. If a student cannot 
answer a question, or an answer is incorrect, then an instructor assessment is equal to zero. If a student answers a 
question corresponding to certain skill correctly, an instructor assessment is equal to fixed initial assessment of 
corresponding skill. It means he or she has developed this skill, if a student answer is not full, an instructor sets 
suitable assessment. As a result, an assessment of a student is made. Assessments of other students from a study 
group are set analogously. 

C. Building skill heterogeneous collaborative groups 

Effective PBCL is caused by the following conditions of building collaborative group: 

• All the students studying a course should have all analytical HOT skills. It allows students to develop needed 
HOT skills as a result of collaboration with other students in a study group 

• Maximum mutual supplementation of skills of students inside a collaborative group. This condition provides 
facilitation of developing skills of the students of the collaborative group through interactions compensating 
for the lack of personal skills 

• Taking into account individual characteristics of students during the formation of collaborative groups. This 
condition allows to remove limitations of collaboration. 

The students will get an opportunity to develop HOT skills through collaboration owing to: 
intra-group interactions among students if cumulative skills of students in collaborative group equal required 
analytical HOT skills; inter-group interactions among students if cumulative skills of students in a collaborative 
group are less then required analytical HOT skills. 

D. Adaptive formation of fixed assessments of HOT skills for second phase of PBL 

The analytical skills were developing on first phase of PBL. Consequently, fixed assessment of this skill type is 
decreased for second phase of PBL. Furthermore, analysis of assessments of analytical HOT skills reveals a lack 
of development certain skills in a study group. It stipulates the need of adaptation of a study group’s 

analytical skills assessments to the process of developing these skills. Adaptation of assessments is realized by 
changing the fixed analytical HOT skills assessments. It is realized for a study group in the following way. If 
examination results revealed a lack of the certain skill development for the study group, then fixed assessment of 
this skill is increased by an instructor. 

E. Assessment of HOT skills development of students after second phase of PBL 

The instructor composes individual lists of control questions on the basis of the complete list for examination of 
analytical skills development. An individual list can contain some questions for examination of a certain 
analytical skill. The individual questions are formed on the basis of skill assessments of a student after first phase 
of PBL. It provides adaptive personalized learning. Next, the instructor composes individual lists of control 
problems based on the complete list. Development of creative skills of a student is assessed by the instructor as a 
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result of solving by him the problems from the individual list. The aggregate of control questions and problems 
from the individual lists should provide examination of all analytical and creative skills. Assessment of HOT 
skills development is realized by comparison of the values of the coefficient of skill development after second 
phase of PBL with the correspondence values of the coefficient of skill development after first phase of PBL. 

F. Assessment of collaborative skills of students 

An assessment of collaborative skills of a student is done by taking into account assessments of his (her) HOT 
skills of a student acquired as a result collaboration. The collaborative group is skill heterogeneous with 
realization of a condition of maximal mutual supplementation of skills of the collaborative group of students. It 
fosters and facilitates skill sharing among students. Assessment of collaborative skills is based on analysis of 
assessments of HOT skills of students after collaborative problem solving. The objective of the analysis is to 
determine a student (students) who obtained a maximal assessment. Such assessment is a result of acquiring 
HOT skills by students due to collaborative problem solving. It can serve as a measure of success of 
collaboration of a student with other students of the collaborative group and also with students of other 
collaborative groups. Hence, a student with a maximal assessment possesses the best collaborative skills. The 
assessments of collaborative skills of students in a study group are calculated proportionally to the assessments 
HOT skills on the basis of fixed assessment of collaborative skills. The combined summative assessments 
characterize outcomes of PBL. These assessments are determined by summation of combined assessments of 
HOT skills and assessments of collaborative skills. 

3. Performance Analysis of Adaptive HOT Skills 

Since long time learners haven’t changed much. They have used the traditional learning method. 

However it has changed the technology tools (Raiyn, 2014). In heterogeneous class room we believe that each 
student has HOT skills, however with different skills level. We have divided HOT skills in three levels, low, 
medium, high. PBL takes into consideration student HOT skills variation between students. To overcome the 
difference between students we introduce personalized learning. Personalized learning means to us that I we 
teaching a student where they are and what they need in order to be successful. In personalized learning process 
the role of teachers, learners will be changed. To transform learning, teachers, learners, learning tools share the 
vision of how they will personalize learning for every student. In personalized learning teachers get new role. 
Instead of delivering knowledge and information, teachers become partners with their learners. Learners first 
understand how they learn best. Then they acquire the skills to choose and use the tools that work best for their 
learning qualities. 

Steps to Personalize Learning 

3.1 Assessment and Evaluation 

In this section we discuss the performance analysis of adaptive higher- order thinking skills. In computer science 
course we have implemented adaptive higher- order thinking assessments. We have used various adaptive 
higher-order thinking assessment to evaluate the creative and analytical thinking skills based on PBL 
assessments as illustrate figures 1 & 2. In the first phase we have measured the high order thinking skills for 
each student based on ordering, selection, evaluation, contracting, and comparison assessments. In the second 
phase we will measure the analytical skills based on flexibility, selection, efficiency, elaboration, and problem 
finding. The students will be divided in subgroups. The groups will be created based on creative skills 
performance for each student in the first phase. 



Figure 1: Process of acquiring analytical skills 
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Figure 2: Process of acquiring critical skills 

Figure 2 illustrates the score of students ordering, selection, and contracting assessments. According to 
Figure 2 appears that the students performance are heterogeneous compared to evaluation, analysis, and 
comparison skills. The HOT skills gab is clear in HOT skills that are focused on selection, ordering, and 
contrasting. 


Analytical Thinking 



■ selecting Bordering ■ contrasting 


adjacent. 


Figure 3a: HOT assessment 

Figure 3a illustrates that evaluation, analysis, and comparison HOT skills by students are high and 


Analytical Skills 



■ comparing ■ evaluation 


Figure 3b: HOT assessments 
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Figure 4 illustrates the approach of creating sub-groups. The groups will be created based on the 
student’s performance in analytical thinking skills assessments. The subgroup’s building is the bridging of the 
increased gap between students. The goal of the subgroups building is to overcome the increasing gab between 
students. 



Figure 4: Sub-group building strategy 



Figure 5: Analytical thinking skills gaps 

Figure 6 illustrates results of critical thinking skills assessments. From the figure we conclude that the 
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performance of students in problem findings skills is weak. The result was expected because the HOT skill of 
PBL was based on defined project and problem for students. 
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Figure 6: Critical thinking skills 

Figure 7 illustrates that student’s critical skills gaps between students have been reduced. Furthermore 
we conclude that the collaboration between students has been increased. Students have worked together to 
accomplish a task of defined problem in sub-groups. The effort of multiple individuals is clearly increased. 




Figure 7: Critical thinking skills gaps 
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4. Conclusion und Future Works 

In this paper we have introduced process of acquiring analytical and creative skills of high order thinking. 
Furthermore we have discussed higher order thinking assessment based on creative skills and analytical skills 
based on flexibility, selection, efficiency, elaboration, and problem finding in order to design automatic 
evaluation for high order thinking skills. Results showed that PBL changes the process of learning in many ways. 
It changes the relationship between learner and teacher, and has an impact at the leaner level. The class has been 
re-organized and re- evaluated according of the principles of PBL. From the teacher’s perspective, PBL based on 
analytical skills demands fundamental and deep reflection on work practices. In the future work we aim to 
develop automatic evaluation of HOT skills assessments based on undefined problem. 
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